The present invention relates to a highly efficient coding method and apparatus used for recording or transmitting picture signals, and more particularly, to a code amount control system that reduces the total code amount of the whole picture to a predetermined value or less, while restraining the deterioration of picture quality.
An example of this type of picture signal coding method is an interframe coding system represented by MPEG (Moving Picture Experts Group). As code amount control in such a coding system, two types of code amount control, that is, (1) intra-GOP interframe code amount control for one GOP (Group of Pictures) and (2) intraframe inter-macro-block code amount control for one frame, are necessary.
In a TM5 system widely known as a test model of the MPEG, for the above (1) intra-GOP interframe code amount control, the target code amount for a coding frame is found from the activity of previously encoded frames and the remaining code amount. Here, the remaining code amount means a code amount obtained by subtracting the generated code amount of encoded frames from a code amount assigned to a GOP to which the coding frame belongs, that is, a code amount assigned to frames after the coding frame.
For the above (2) intraframe inter-macro-block code amount control, the quantization parameter for a coding macro block (hereinafter referred to as a scaling factor) is found from the relationship between the generated code amount up to the present time and the target code amount. This system is called a feedback system.
On the contrary, in a feedforward system, which is also well known for the intraframe inter-macro-block code amount control, a code amount is pre-calculated by using all available scaling factors (in the MPEG 2, 31 scaling factors can be used) and the scaling factor that is the closest to the target code amount among the obtained 31 code amounts is selected as the final scaling factor.
The scaling factor is a parameter used in quantization. One scaling factor is provided corresponding to one macro block. When the value of the scaling factor is small, the macro block is quantized in a smaller quantization step. On the contrary, when the value of the scaling factor is large, the macro block is quantized in a larger quantization step.
FIG. 16 shows a block diagram of a picture signal coding device using a conventional intra-GOP interframe code amount control system. In FIG. 16, 800 denotes a motion vector detection portion, 801 denotes a differential picture generation portion, 802 denotes an activity calculation portion, 803 denotes a target code amount determination portion, 804 denotes a coding portion, 805 denotes an assigned code amount updating portion, and 806 denotes a local decoder.
A picture signal is inputted to the motion vector detection portion 800 and the differential picture generation portion 801. When the inputted picture type is a P type or B type picture, the motion vector detection portion 800 detects motion vectors with respect to reference picture(s) stored in the memory and outputs the detected motion vectors. When the inputted picture is an I type picture, the motion vector detection portion 800 does not detect motion vectors. The motion vectors outputted from the motion vector detection portion 800 are inputted to the differential picture generation portion 801.
When the inputted picture type is a P type or B type picture, the differential picture generation portion 801 generates a predicted picture from the inputted motion vectors and decoded reference picture(s) inputted from the local decoder 806 and calculates the differences between the predicted picture and the inputted picture. The differential picture outputted from the differential picture generation portion 801 is inputted to the coding portion 804.
If there is a frame that is the same picture type in the previously encoded frames, the target code amount determination portion 803 determines a target code amount from the activity of the encoded frame and the remaining code amount. If there is no frame of the same picture type, the target code amount determination portion 803 sets a specific initial value as the target code amount. The target code amount is inputted to the coding portion 804.
The coding portion 804 encodes the inputted differential picture by the inputted target code amount and outputs encoded data and an average scaling factor. The encoded data is inputted to the activity calculation portion 802, the assigned code amount updating portion 805, and the local decoder 806. The average scaling factor is inputted to the activity calculation portion 802.
The activity calculation portion 802 calculates a generated code amount from the inputted encoded data, calculates an activity from the inputted average scaling factor and the calculated generated code amount, and updates the activity of the frame of the same picture type as the coding frame to the calculated activity.
The assigned code amount updating portion 805 calculates a generated code amount from the inputted encoded data and updates the assigned code amount. The local decoder 806 decodes the inputted encoded data and generates a decoded picture.
FIG. 17 shows a block diagram of the coding portion 804. In FIG. 17, 810 denotes a macro-blocking portion, 811 denotes an orthogonal transformation portion, 812 denotes a quantization portion, 813 denotes an importance calculation portion, 814 denotes a scaling factor determination portion, and 815 denotes a variable length coding portion. A conventional intraframe inter-macro-block code amount control system will be described with reference to this figure.
A differential picture inputted to the coding portion is inputted to the macro-blocking portion 810. The macro-blocking portion 810 performs the process of blocking and macro-blocking the inputted differential picture. The macro-blocked data is inputted to the orthogonal transformation portion 811 and the importance calculation portion 813.
The orthogonal transformation portion 811 performs orthogonal transformation for each block and outputs transformation coefficients. The transformation coefficients are inputted to the quantization portion 812. The importance calculation portion 813 calculates an importance for each macro block and outputs the importance data indicating the importance of the macro block.
The importance data is inputted to the scaling factor determination portion 814. The scaling factor determination portion 814 determines a scaling factor from the importance data of the coding macro block and the remaining code amount. Here, the remaining code amount means a code amount obtained by subtracting the sum of the generated code amounts of macro blocks previously encoded from the target code amount set for the frame, that is, a code amount assigned to the remaining macro blocks including the coding macro block.
The scaling factor outputted from the scaling factor determination portion 814 is inputted to the quantization portion 812. The quantization portion 812 quantizes the transformation coefficients provided from the orthogonal transformation portion 811 by the inputted scaling factor and outputs quantized data. The quantized data is inputted to the variable length coding portion 815.
The variable length coding portion 815 performs the variable length coding of the inputted quantized data and outputs encoded data. The encoded data is inputted to the scaling factor determination portion 814. The scaling factor determination portion 814 calculates the generated code amount of the encoded macro block based on the inputted encoded data and updates the remaining code amount.
FIG. 18 is a block diagram of another configuration of the conventional coding portion 804. In FIG. 18, 820 denotes a macro-blocking portion, 821 denotes an orthogonal transformation portion, 822 denotes a quantization portion, 823 denotes an importance calculation portion, 824 denotes a code amount calculation portion, 825 denotes a final scaling factor determination portion, and 826 denotes a variable length coding portion. A feedforward system, which is another intraframe inter-macro-block code amount control system, will be described with reference to FIG. 18. Only the coding portion will be described as in the above explanation with reference to FIG. 17.
A differential picture inputted to the coding portion is inputted to the macro-blocking portion 820. The macro-blocking portion 820 performs the process of blocking and macro-blocking the inputted differential picture. The macro-blocked data is inputted to the orthogonal transformation portion 821 and the importance calculation portion 823.
The orthogonal transformation portion 821 performs orthogonal transformation for each block and outputs transformation coefficients. The transformation coefficients are inputted to the quantization portion 822 and the code amount calculation portion 824. The importance calculation portion 823 calculates an importance for each macro block and outputs importance data indicating the importance of the macro block. The importance data is inputted to the code amount calculation portion 824 and the quantization portion 822.
The code amount calculation portion 824 has 31 code amount estimation portions. Each code amount estimation portion performs the quantization and variable length coding of the inputted transformation coefficients based on a specific scaling factor and a scaling factor generated from the importance data and calculates the generated code amount of one frame. The generated code amounts according to 31 scaling factors outputted from the code amount calculation portion 824 are inputted to the final scaling factor determination portion 825.
The final scaling factor determination portion 825 determines a scaling factor providing a generated code amount that is the closest to the target code amount of the coding frame among the inputted 31 generated code amounts as a final scaling factor and outputs the final scaling factor. The final scaling factor is inputted to the quantization portion 822.
The quantization portion 822 quantizes the macro block of the coding frame based on the inputted final scaling factor and importance data and outputs quantized data. The quantized data is inputted to the variable length coding portion 826. The variable length coding portion 826 performs the variable length coding of the inputted quantized data and outputs encoded data.
A problem of the conventional intra-GOP interframe code amount control systems as described above is that since the activity of the frames previously encoded is used to determine the target code amount of the coding frame, the assignment of the code amount does not reflect the characteristics of the picture of the coding frame. Furthermore, in the intraframe inter-macro-block code amount control system, the scaling factor used for the quantization of the coding macro block does not reflect the characteristics of the coding macro block sufficiently.
In addition, the conventional feedforward type intraframe inter-macro-block code amount control system enables control that takes into account the characteristics of the coding macro block, but this system requires a very large scale circuit and a very large number of operations.
A coding method according to the present invention is adapted to control code amounts with high precision according to the characteristics of pictures by improving first and second conventional coding methods as defined below.
The first conventional coding method comprises the steps of defining N continuous frames as one coding control unit, performing intraframe coding of a first frame of the coding control unit without using a difference between frames, coding an Nth frame using a difference between the Nth frame and the first frame, and coding a second frame to a (Nxe2x88x921)th frame using both difference between the second to the (Nxe2x88x921)th frame and the first frame and difference between the second to the (Nxe2x88x921)th frame and the Nth frame.
The second conventional coding method comprises the steps of defining N continuous frames as one coding control unit, performing intraframe coding of a Kth frame of the coding control unit without using a difference between frames, coding an Nth frame using a difference between the Nth frame and the Kth frame, coding a first frame to a (Kxe2x88x921)th frame using both difference between the first to the (Kxe2x88x921)th frame and the Nth frame of the latest previous coding control unit and difference between the first to the (Kxe2x88x921)th frame and the Kth frame of the current coding control unit, and coding a (K+1)th frame to a (Nxe2x88x921)th frame using both difference between the (K+1)th to the (Nxe2x88x921)th frame and the Kth frame of the current coding control unit and difference between the (K+1)th to the (Nxe2x88x921)th frame and the Nth frame of the current coding control unit,
According to a first aspect of the coding method of the present invention, in the first conventional coding method, target code amounts of all frames of the coding control unit are set considering the difference between the Nth frame and the first frame as an index.
According to a second aspect of the coding method of the present invention, in the second conventional coding method, target code amounts of all frames of the current coding control unit are set considering the difference between the Nth frame and the Kth frame as an index.
According to a third aspect of the coding method of the present invention, in the first conventional coding method, a scene change in the coding control unit is detected based on the motion amount of the Nth frame from the first frame and the motion amounts of the second to the (Nxe2x88x921)th frame from the first frame, and target code amounts of all frames of the coding control unit are set considering the result of the scene change detection as an index.
According to a fourth aspect of the coding method of the present invention, in the second conventional coding method, a scene change in the coding control unit is detected based on the motion amount of the Nth frame from the Kth frame and the motion amounts of the first to the (Kxe2x88x921)th frame and the (K+1)th to the (Nxe2x88x921)th frame from the Kth frame, and target code amounts of all frames of the coding control unit are set considering the result of the scene change detection as an index.
According to a fifth aspect of the coding method of the present invention, in the first conventional coding method, the degree of difficulty of coding picture signals in the coding control unit is detected from an activity calculated with respect to the first frame, and target code amounts of all frames of the coding control unit are set considering the degree of difficulty and the difference between the Nth frame and the first frame as indices.
According to a sixth aspect of the coding method of the present invention, in the second conventional coding method, the degree of difficulty of coding picture signals in the coding control unit is detected from an activity calculated with respect to the Kth frame, and target code amounts of all frames of the coding control unit are set considering the degree of difficulty and the difference between the Nth frame and the Kth frame as indices.
According to the coding methods as described above, the code amounts can be controlled according to the characteristics of pictures. Especially, changes in pictures in a GOP can be predicted according to the differential information of the P picture. Thus, even in scene-changed pictures or pictures whose motion changes greatly, which are conventionally a problem, code amounts can be assigned according to the characteristics of pictures. Therefore, the process of the present invention can achieve a higher picture quality than conventional processes at the same bit rate. Furthermore, the process of the present invention can achieve the same picture quality at a bit rate lower than that of the conventional processes.
According to a first aspect of a coding device of the present invention, the coding device comprises a unit division means for divining inputted pictures into coding control units comprising one or plurality of frames that are to be subjected to a process selected from the group consisting of an intraframe coding process, an interframe forward coding process, and an interframe bidirectionally predictive coding process; an activity calculation means for calculating a frame activity from each frame of the inputted pictures and calculating a unit activity for each unit from the frame activities of the frames that belong to the unit; a target code amount determination means for determining the target code amount of a coding frame based on the frame activity of the coding frame, the unit activity of a coding unit to which the coding frame belongs, and the average unit activity of previous units; and a coding means for coding the coding frame with the target code amount to generate encoded data.
According to the above aspect, the inputted pictures are controlled for each unit, so that code amounts can be controlled according to the characteristics of pictures. It is preferable that the frame activity calculated for each frame is multiplied by one of three weighting constants predetermined according to whether the coding frame is an I, P or B picture type and that the resulting value is used as a new frame activity. Code amounts can be controlled with high precision according to the characteristics of pictures by performing a weighting operation of an activity according to the picture type.
According to a second aspect of the coding device of the present invention, the coding device comprises an activity function generation means for calculating an activity for each macro block in each frame of inputted pictures, generating an activity function indicating the accumulation of the activity for each macro block in a coding frame, and calculating the frame activity of the coding frame; a target code amount determination means for determining the target code amount of the coding frame by considering at least one of the remaining code amount at the time of coding the coding frame, the generated code amount of previously encoded frames, the activity of the previously encoded frames, the occupancy of a buffer, and the activity of the coding frame; a reference scaling factor calculation means for calculating the reference scaling factor of the coding frame by using at least one of the scaling factor of the previously encoded frames, the generated code amount of the previously encoded frames, the frame activity of the previously encoded frames, the frame activity of the coding frame, and the target code amount; an importance calculation means for detecting a visual importance for each macro block; and a coding means for generating a prediction model based on the activity function and the target code amount, coding the coding frame macro block by macro block by using the prediction model, the reference scaling factor, and the importance, and generating encoded data.
According to the above aspect, coding is performed along the prediction model generated from the activity reflecting the characteristics of the picture type and the reference scaling factor is controlled by the difference between the prediction model and the generated code amount. Therefore, code amounts can be controlled with high precision according to the characteristics of pictures.
According to a third aspect of the coding device of the present invention, the coding device comprises an activity function generation means for calculating an activity for each macro block in each frame with respect to inputted pictures, generating an activity function indicating the accumulation of the activity for each macro block in a coding frame, and calculating the frame activity of the coding frame; a target code amount determination means for determining the target code amount of the coding frame by using at least one of the remaining code amount at the time of coding the coding frame, the generated code amount of previously encoded frames, the activity of the previously encoded frames, the occupancy of a buffer, and the activity of the coding frame; a reference scaling factor calculation means for calculating the reference scaling factor of the coding frame by using at least one of the scaling factor of the previously encoded frames, the generated code amount of the previously encoded frames, the frame activity of the previously encoded frames, the frame activity of the coding frame, and the target code amount; an importance calculation means for detecting a visual importance for each macro block; and a coding means for generating a prediction model based on the activity function and the target code amount, setting a region with a specific width with respect to the prediction model, coding the coding frame macro block by macro block by using the reference scaling factor and the importance, correcting at least one of the reference scaling factor, the prediction model, the target code amount, and the region according to a deviation amount when the accumulation of a generated code amount for each macro block deviates from the region, and generating encoded data.
According to the above aspect, the region with a specific width is set with respect to the prediction model generated from the activity reflecting the characteristics of the picture, coding is performed along the prediction model, and the reference scaling factor is controlled according to the deviation amount when the generated code amount deviates from the region. Therefore, code amounts can be controlled with high precision according to the characteristics of pictures.
Various preferable embodiments for implementing the second and third aspect of the coding device according to the present invention will be described below.
First, the coding means sets a check point for each integer macro blocks and determines whether the accumulation of the generated code amount for each macro block deviates from the region at the check point or not. By setting a check point for each integer macro blocks, the fluctuation of the reference scaling factor can be reduced. Therefore, stable regenerated pictures can be provided while performing control with high precision according to the characteristics of pictures.
In addition, the importance calculation means calculates the importance for each orthogonal transformation block, calculates the sum of absolute values of AC components in orthogonal transformation data of the orthogonal transformation block when the coding system of a coding block is an interframe coding system and when the coding frame is a B picture, and sets the scaling factor of the orthogonal transformation block smaller than the reference scaling factor if the sum of the absolute values is larger than a threshold. By protecting regions for a moving object and regions whose motion vector is large, code amounts can be controlled with high precision according to the characteristics of pictures.
Furthermore, the importance calculation means calculates the importance for each orthogonal transformation block, calculates the sum of absolute values of AC components in orthogonal transformation data of the orthogonal transformation block when the coding system of a coding block is an interframe coding system and when the coding frame is a B picture, and sets all orthogonal transformation data of the orthogonal transformation block to 0 if the sum of the absolute values is smaller than a threshold. Reproducing still regions and regions where motion vector is effective only by motion compensation, large code amounts can be assigned to other regions. Therefore, code amounts can be controlled with high precision according to the characteristics of pictures.
Moreover, the importance calculation means calculates the importance for each orthogonal transformation block; calculates the number of sampling values having a larger value than a threshold among the sampling values of a Y-R component block in the orthogonal transformation block with respect to an inputted picture for which a differential operation for motion compensation is not performed, when the coding system of a coding block is an interframe coding system; and sets the scaling factor of the orthogonal transformation block smaller than the reference scaling factor if the number of the sampling values is larger than a predetermined value. By protecting a red color having a high chroma, the picture quality can be improved.
In addition, the importance calculation means calculates the importance for each orthogonal transformation block; calculates the number of sampling values having a larger value than a threshold among the sampling values of a Y-B component block in the orthogonal transformation block with respect to an inputted picture for which a differential operation for motion compensation is not performed, when the coding system of a coding block is an interframe coding system; and sets the scaling factor of the orthogonal transformation block smaller than the reference scaling factor if the number of the sampling values is larger than a predetermined value. By protecting a blue color having a high chroma, the picture quality can be improved.
Furthermore, the importance calculation means calculates the importance for each orthogonal transformation block; calculates the number of sampling values having a value in the range of a first threshold to a second threshold with respect to a Y component block in a coding macro block and defines the number of the sampling values as a first detected value, calculates the number of sampling values having a value in the range of a third threshold to a fourth threshold with respect to a Y-R component block in the coding macro block and defines the number of the sampling values as a second detected value, and calculates the number of sampling values having a value in the range of a fifth threshold to a six threshold with respect to a Y-B component block in the coding macro block and defines the number of the sampling values as a third detected value, with respect to an inputted picture for which a differential operation for motion compensation is not performed, when the coding system of a coding block is an interframe coding system; and sets the scaling factor of a coding macro block smaller than the reference scaling factor when the first detected value is larger than a first number, when the second detected value is larger than a second number, and when the third detected value is larger than a third number. By detecting a specific color, for example, a skin color, and protecting the color, the picture quality can be improved.
Moreover, the importance calculation means calculates the importance for each orthogonal transformation block; calculates the number of sampling values in which the absolute value of a difference between a sampling value of a Y-R component block and a sampling value of a Y-B component block is larger than a threshold with respect to the Y-B component block and the Y-R component block in a coding macro block, with respect to an inputted picture for which a differential operation for motion compensation is not performed, when the coding system of a coding block is an interframe coding system; and sets the scaling factor of a coding macro block smaller than the reference scaling factor if the number of the sampling values is larger than a predetermined value. By protecting red and blue colors having a high chroma, the picture quality can be improved.
In addition, the importance calculation means calculates the importance for each orthogonal transformation block; calculates the activity of the orthogonal transformation block with respect to an inputted picture for which a differential operation for motion compensation is not performed and calculates the sum of absolute values of AC components in the orthogonal transformation block with respect to a differential picture for which a differential operation for motion compensation is performed when the coding system of a coding block is an interframe coding system; and sets the scaling factor of the orthogonal transformation block larger than the reference scaling factor when the activity is larger than a first threshold and the sum of the absolute values of the AC components is larger than a second threshold. By not protecting regions where visual deterioration is unnoticeable and providing large code amounts to other regions, code amounts can be controlled with high precision according to the characteristics of pictures.
Furthermore, the importance calculation means calculates the importance for each macro block, and sets the scaling factor of a coding macro block larger than the reference scaling factor when the coding macro block is located at one of the upper end, lower end, left end and right end of the screen. By not protecting regions where visual deterioration is unnoticeable and providing large code amounts to other regions, code amounts can be controlled with high precision according to the characteristics of pictures.
Moreover, the importance calculation means calculates the importance for each orthogonal transformation block; and sets the scaling factor of a coding macro block smaller than the standard scaling factor when the coding system of a coding block is an interframe coding system, when types of the motion vector of the coding macro block and the motion vectors of macro blocks around the coding macro block are detected, and when the type of the motion vector of the coding macro block differs from the type of the motion vectors of the macro blocks around the coding macro block. By protecting regions for a moving object and regions whose motion vector is large, code amounts can be controlled with high precision according to the characteristics of pictures.
In addition, a motion vector detection portion properly switches a plurality of predictive operation methods with respect to an inputted picture, and wherein a code amount control means calculates the importance for each macro block, and sets the importance of a coding macro block high when the coding system of a coding block is an interframe coding system and when the type of a first motion vector detected for each macro block differs from the type of a second motion vector detected for each super block comprising a plurality of macro blocks. By protecting regions for a moving object and regions whose motion vector is large, code amounts can be controlled with high precision according to the characteristics of pictures.
Furthermore, the importance calculation means detects importances with respect to T orthogonal transformation blocks that belong to the macro block, and defines the highest importance among the detected T importances as the importance of the macro block.
Moreover, the importance calculation means detects the importance for each orthogonal transformation block, and sets all data of the block after orthogonal transformation to 0 and sets the activity of the orthogonal transformation block to 0 when the block is determined as unimportant.